MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization

We describe the MADA+TOKAN toolkit, a versatile and freely available system that can derive extensive morphological and contextual information from raw Arabic text, and then use this information for a multitude of crucial NLP tasks. Applications include high-accuracy part-of-speech tagging, diacritization, lemmatization, disambiguation, stemming, and glossing. MADA operates by examining a list of all possible analyses for each word, and then selecting the analysis that matches the current context best by means of support vector machine models classifying for 19 distinct, weighted morphological features. The selected analyses carry complete diacritic, lexemic, glossary and morphological information; thus all disambiguation decisions are made in one step. TOKAN takes the information provided by MADA to generate tokenized output in a wide variety of customizable formats. MADA, TOKAN and their support utilities are highly configurable, allowing users to extract and manipulate the exact information that they require. In this paper we describe the features and capabilities of MADA+TOKAN, detail recent improvements, and provide examples of the toolkit’s use.